xc2xa71.1 Field of the Invention
The present invention concerns a user interface to objects, such as informational objects for example, which are stored on, or accessed via, a machine such as a computer for example. The present invention exploits the spatial memory of people.
xc2xa71.2 Related Art
A brief history of user interfaces is provided below. First, known user interfaces are introduced in xc2xa71.2.1. Then, user interfaces which facilitate information access, rather than information processing, are introduced in xc2xa71.2.2. Finally, needs of users which have not yet been met by user interfaces are listed in xc2xa71.2.3.
The way in which people interact with computing machines has evolved over the last 50 or so years. Initially, these machines were typically used as information processors, and in particular, for performing mathematical operations on numbers. People interacted with such early computing machines by punching and ordering cards to effect a sequence of commands. In later computing machines, also typically used to perform mathematical operations on numbers, people interacted with such computing machines by setting switches and viewing light emitting diodes to enter commands. With the advent of the desktop personal computer, people-machine interaction evolved from the use of a keyboard to enter lines of commands, discussed in xc2xa71.2.1.1 below, to the use of a keyboard and mouse to manipulate icon metaphors of the real world, discussed in xc2xa71.2.1.2 below. cl xc2xa71.2.1.1 Command Lines
Early personal computers were also used to perform mathematical operations, from engineering applications to accounting applications (e.g., spreadsheets). In addition, such early personal computers were used to enter, store, and manipulate information, such as with word processing applications for example, and to effectively access stored information, such as with relational database applications for example. People typically interacted with such computers by entering commands, in accordance with fairly rigid syntactical rules, or by entering data via a keyboard and viewing results via a video monitor. Unfortunately, since the syntactical rules for interacting with a personal computer were typically not intuitive, people would have to invest a fairly substantial amount of time to master effective interaction with the computer. Until they mastered the syntactical rules, people would often become frustrated when using computers. Casual users would often conclude that learning to interact with computers would not be worth their time. Thus, computer use was usually limited to professionals (e.g., accountants, engineers, and scientists), who needed the computational power of personal computers, and to hobbyists.
The advent of graphical user interfaces (or xe2x80x9cGUIsxe2x80x9d) provided a more intuitive way for people to interact with computers. The casual user no longer needed to learn syntactical rules and enter sequences of commands. As personal computers penetrated business to an ever greater degree, probably due to popular and time saving word processing and spreadsheet applications, operating systems began to employ user interfaces which used a office metaphor which included documents, folders, filing cabinets, trash cans, telephone directories, etc. These so-called desktop GUIs have served their users well over the past decade or so. However, in addition to using computers for data entry, storage and manipulation, people are using computers for access to information to an ever increasing degree. This recent trend is discussed in xc2xa71.2.2 below.
In recent decades, and in the past five (5) to ten (10) years in particular, computers have become interconnected by networks by an ever increasing extent; initially, via local area networks (or xe2x80x9cLANsxe2x80x9d), and more recently via LANs, private wide area networks (or xe2x80x9cWANsxe2x80x9d) and the Internet. The proliferation of networks, in conjunction with the increased availability of inexpensive data storage means, has afforded computer users unprecedented access to a wealth of content. Such content may be presented to a user (or xe2x80x9crenderedxe2x80x9d) in the form of text, images, audio, video, etc.
The Internet is one means of inter-networking local area networks and individual computers. The popularity of the Internet has exploded in recent years. Many feel that this explosive growth was fueled by the ability to link (e.g., via Hyper-text links) resources (e.g., World Wide Web pages) so that users could seamlessly transition from various resources, even when such resources were stored at geographically remote resource servers. More specifically, the Hyper-text markup language (or xe2x80x9cHTMLxe2x80x9d) permits documents to include hyper-text links. These hyper-text links, which are typically rendered in a text file as text in a different font or color, include network address information to related resources. More specifically, the hyper-text link has an associated uniform resource locator (or xe2x80x9cURLxe2x80x9d) which is an Internet address at which the linked resource is located. When a user activates a hyper-text link, for example by clicking a mouse when a displayed cursor coincides with the text associated with the hyper-text link, the related resource is accessed, downloaded and rendered to the user. The related resource may be accessed by the same resource server that provided the previously rendered resource, or may be accessed by a geographically remote resource server. Such transiting from resource to resource, by activating hyper-text links for example, is commonly referred to as xe2x80x9csurfingxe2x80x9d.
Although people continue to use computers to enter information, manipulate information, and store information, in view of the foregoing developments people are using computers to access information to an ever increasing extent. In a departure from the past, the information people want to access is often not created by them (which would typically reside on the person""s desktop computer), or even by a company or group to which that person belongs (which would typically reside on a storage server, accessible via a local area network). Rather, given the world wide breadth of the Internet, the information people want to access is usually created by unrelated third parties (or content providers). Unfortunately, GUIs using a desktop metaphor are not particularly well suited for such information access.
New GUIs should therefore help people find information that they want, or that they might want. Unfortunately, the very vastness of available data can overwhelm a user; desired data can become difficult to find and search heuristics employed to locate desired data often return unwanted data (also referred to as xe2x80x9cnoisexe2x80x9d).
Various concepts have been employed to help users locate desired data. In the context of the Internet for example, some services have organized content based on a hierarchy of categories. A user may then navigate through a series of hierarchical menus to find content that may be of interest to them. An example of such a service is the YAHOO(trademark) World Wide Web site on the Internet. Unfortunately, content, in the form of Internet xe2x80x9cweb sitesxe2x80x9d for example, must be organized by the service and users must navigate through a predetermined hierarchy of menus. If a user mistakenly believes that a category will be of interest or include what they were looking for, but the category turns out to be irrelevant, the user must backtrack through one or more hierarchical levels of categories.
Again in the context of the Internet for example, some services provide xe2x80x9csearch enginesxe2x80x9d which search database content or xe2x80x9cweb sitesxe2x80x9d pursuant to a user query. In response to a user""s query, a rank ordered list, which includes brief descriptions of the uncovered content, as well as a hypertext links (text, having associated Internet address information, which, when activated, commands a computer to retrieve content from the associated Internet address) to the uncovered content is returned. The rank ordering of the list is typically based on a match between words appearing in the query and words appearing in the content. Unfortunately, however, present limitations of search heuristics often cause irrelevant content (or xe2x80x9cnoisexe2x80x9d) to be returned in response to a query. Again, unfortunately, the very wealth of available content impairs the efficacy of these search engines since it is difficult to separate irrelevant content from relevant content.
The foregoing means permit users to find content of interest to them, or to deliver content that may be of interest to a user. Unfortunately, neither of the foregoing means is particularly well suited when the user wants to xe2x80x9cgo backxe2x80x9d to (or xe2x80x9crelocatexe2x80x9d) information (or content), to revisit that information or to revisit a familiar content provider to view new information (or content). More specifically, a person typically does not want to go through steps of navigating through a hierarchy of menus, or entering a search query to get to favored content or favored content providers.
In view of the shift towards using computers to access information, a number of user interfaces has been developed to help people revisit or relocate information or content providers. A sample of these user interfaces, as well as perceived limitations of such user interfaces, is presented below.
Some so-called xe2x80x9cInternet browserxe2x80x9d program services, such as Microsoft""s Internet Explorer(trademark) for example, permit people to create lists of favorite Internet locations (as located by a Uniform Resource Locator or xe2x80x9cURLxe2x80x9d) represented by bookmarks. Unfortunately, as the number of bookmarks in a list increases, a person""s ability to find a desired bookmark becomes more difficult. Although people can arrange and organize such lists of bookmarks into hierarchies, this requires some effort (mental effort is also referred to as xe2x80x9ccognitive loadxe2x80x9d) to organize and maintain the hierarchy. Moreover, the use of hierarchies does not fully exploit the spatial memory (This concept has also been referred to as xe2x80x9cwhere it is is what it isxe2x80x9d.) of people.
Recognizing the trend towards the use of computers for information access, the article: Robertson, et al., xe2x80x9cThe Next Generation GUIs: Information Visualization Using 3D Interactive Animation,xe2x80x9d Communications of the ACM, Vol. 35, No. 4, pages 57-71 (April 1993) (hereinafter referred to as xe2x80x9cthe Information Visualizer articlexe2x80x9d) discusses various proposals for an xe2x80x9cInformation Workspacexe2x80x9d. More specifically, a three-dimensional rooms metaphor, three dimensional hierarchical cone trees, and perspective walls are discussed as means to present information to users. Each of these techniques is introduced below.
Although the three-dimensional room metaphor exploits, at least to some degree, a person""s spatial memory, the person has to manipulate objects or move to disambiguate images and reveal hidden information.
Although the three-dimensional hierarchical cone trees are useful for visualizing large amounts (e.g., 600 directories of 10,000 files) of hierarchical data, they lend themselves to predetermined hierarchies such as file directories, organizational structure, etc. That is, it is believed that the strength of three-dimensional hierarchical cone trees lies in presenting information, not building or updating hierarchies of information.
The perspective wall permits information, having some sort of linear relationship or thread, to be presented in the relatively narrow aspect ratio of a typical video monitor. When an item is selected, the wall moves the item to a center portion of the wall as if it were a sheet in a player piano reel. Its intuitive three-dimensional metaphor allows smooth transitions among views, thereby helping a user to perceive object consistency. Files may be classified by their modification date. Although the perspective wall technique lends itself to information having a linear (e.g., timeline) thread, this technique is less useful for other types of information, or for information in which a linear thread is unimportant. In fact, to have maximum impact, the perspective wall should be combined with an information retrieval technique for highlighting similar or related items to a selected item. Moreover, a user is confined to relating information to some type of a linear thread.
Apart from the limitations of the three-dimensional hierarchical cone tree and perspective wall techniques introduced above, in these techniques, there is little user choice about where an informational object is located in the user interfacexe2x80x94it either has a spot in a hierarchy or a spot on a linear thread. Although it is true that a user can define a hierarchical structure or determine what linear thread the informational objects are to be related, once the structure or thread is determined, user choice about where to locate the information object is removed.
Another article, Card, et al., xe2x80x9cThe WebBook and Web Forager: An Information Workspace for the World-Wide Web,xe2x80x9d Proceedings of CHI ""96, pp. 111-117 (Apr. 13-18, 1996) (hereafter referred to as xe2x80x9cthe WebBook articlexe2x80x9d) also recognized the trend towards the use of computers for information access. Focusing on the use of the Internet, the WebBook article noted, among other things, that web pages are often hard to find, users get lost and have difficulty relocating pages, and users have difficulty organizing found pages. The article then discussed a WebBook(trademark) type electronic book which simulates, in three dimensions, a physical book and includes pages, each of which correspond to a web page. Hypertext links are color coded to indicate whether the referenced web page is within the currently opened book or not. If the web page referenced by the link is in the book, activating the hypertext link will effect an animation of flipping pages to the desired web page. If the web page referenced by the link is not in the book, activating the hypertext link will close the currently opened WebBook type book and, if the web page is in another WebBook type book on a simulated bookshelf, will open that WebBook type book to the selected web page. The user can flip or ruffle through the pages of the WebBook type book using various input techniques. Portions of interest on a web page may be inspected with panning and zooming operations. The Document Lens feature described in the WebBook article is related to U.S. Pat. No. 5,670,984 to Robertson (incorporated herein by reference).
Although the WebBook type book uses screen space efficiently and uses a familiar metaphor, it has a number of limitations. First, the book metaphor limits the number of web pages that can be seen at one timexe2x80x94most pages are occluded. Second, the book metaphor does not exploit spatial memory. That is, it is believed that user will only remember the ordered pages in a linear (or one-dimensional) manner.
The WebBook article also discusses a Web Forager which embeds the WebBook type book and other objects in a hierarchical three-dimensional workspace. The workspace includes four (4) hierarchical levels; namely a focus place, an immediate memory space, a secondary storage area, and a tertiary storage area. In the focus place, objects are rendered in full size and can be acted upon by the user. The immediate memory space uses the metaphor of a desk top. The secondary storage area uses several tiers in Z-space, in which objects can be moved. Finally the tertiary storage area uses a bookshelf metaphor. A book selected from the bookshelf will come up to the focus place and an object then in the focus place will be moved into the intermediate memory space.
Although the Web Forager exploits, at least to some extent, spatial memory, particularly in the secondary storage area, the interaction between the focus place, immediate memory place, secondary storage area, and tertiary storage area may not be immediately obvious to new users. Further, if more than about 30 objects are placed in the secondary storage place, objects may become occluded. Finally, the use of tiers in the Z-dimension of the secondary storage place limits the exploitation of a user""s spatial memory.
Noting the concept of xe2x80x9cwhat it is is where it isxe2x80x9d, the article, Ballay, xe2x80x9cDesigning Workscape(trademark): An Interdisciplinary Experiencexe2x80x9d, Human Factors in Computing Systems, CHI ""94, pp. 10-15 (April 1994) (hereafter referred to as xe2x80x9cthe Workscape articlexe2x80x9d), discusses a three-dimensional user interface for managing documents. Workscape permits users to drag documents in the X-Y plane, and also push and pull documents in the Z-dimension.
Although Workscape exploits spatial memory, it is apparently not concerned with the problem of object occlusion. Furthermore, it is believed that separate user operations for (i) dragging an object in the X-Y plane, and (ii) pushing and pulling documents in the Z dimension will be cumbersome for users in practice.
The product MAPA, offered by Dynamic Diagrams of Providence, R.I., uses a three-dimensional display to show the organization of web pages at a web site. As a web page is selected, its children in the web site hierarchy are displayed. The pages are represented by sheets. As a user""s cursor hovers over a page, a pop-up title is provided. Like the hierarchical cone trees discussed in xc2xa71.2.2.1.2 above, although MAPA may be useful for visualizing large amounts of hierarchical data, it lends itself to predetermined hierarchies (i.e., web pages of a web site). That is, it is believed that the strength of MAPA is presenting information, not building or updating hierarchies of information.
As discussed above, there exists a need for a user interface, and in particular a graphical user interface, to information or content. Such a user interface should exploit spatial memory. For example, the user interface should simulate three dimensions, and should permit continuous movement in the simulated space, to exploit spatial memory to the fullest extent. Other means or cues for reinforcing the three-dimensional environment queues should be used. The user interface should also be intuitive to minimize the time needed for a user to become familiar with it. For example, the user interface should not be cumbersome or require too many different types of inputs to manipulate objects. Finally, the user interface should provide intelligent help to the user.
The present invention provides a user interface, and in particular a graphical user interface, to organize and access information or content (also referred to as an xe2x80x9cobjectxe2x80x9d). The present invention permits a user to view and organize all objects and to edit or otherwise work on a selected object by, for example, representing, graphically, objects or content with a low resolution image which can be added, moved, or deleted from a simulated three-dimensional environment on the user""s video monitor. The present invention may use pop-up title or information bars for permitting a user to discern more about the object represented by a low resolution image. The present invention may use higher resolution image representations of objects, or xe2x80x9clivexe2x80x9d objects loaded within an associated application, for editing or otherwise working on a selected object. Other visual representations of objects may be used, such as schematic or iconic representations of the content of each object for example. Any and all visual representations of objects may be referred to as xe2x80x9cthumbnailsxe2x80x9d in the following.
The user interface of the present invention exploits spatial memory by, for example, simulating a plane located and oriented in three-dimensional space, or other three-dimensional landscape on which the object thumbnails may be manipulated. The plane or landscape may include visual landmarks for enhancing a user""s spatial memory. As the object thumbnails are moved about the landscape, the present invention may employ perspective views (perceived image scaling with distance), partial image occlusion, shadows, and/or spatialized audio to reinforce the simulated three-dimensional plane or landscape. Other audio cues may be used to indicate proximal relationships between object thumbnails, such as when an object thumbnail being xe2x80x9cmovedxe2x80x9d is close to a pre-existing cluster of object thumbnails. An ancillary advantage of using a simulated three-dimensional landscape is that more objects can be represented, at one time, on a single display screen.
The user interface of the present invention is intuitive and minimizes the time needed for a user to become familiar with it. The user interface of the present invention is neither cumbersome, nor does it require too many different types of inputs to manipulate the object xe2x80x9cthumbnailsxe2x80x9d. For example, the present invention may use inputs from a familiar input device such as a mouse or pointer to manipulate the object thumbnails. To minimize the number of different types of inputs required to manipulate the object thumbnails, the present invention may map two-dimensional inputs, such as moving a mouse on a mouse pad, to a three-dimensional movement on the simulated three-dimensional display. The location of the objects may be constrained to the plane or landscape. The present invention may also prevent one object thumbnail (or landscape feature) from totally occluding another object thumbnail so that the latter is not totally hidden from the user. To further reinforce the simulated three-dimensional environment, the present invention may simulate head motion parallax. In this regard, the present invention may use a camera (or head or body mounted equipment) to detect the position or, alternatively, the orientation, of a user""s head relative to a video monitor on which the user interface is rendered.
Finally, the user interface of the present invention can provide intelligent help to the user. For example, the present invention may cluster, by rendering a visual boundary for example, object thumbnails which are arranged, by the user, relatively close to one another. Further, the present invention may employ a matching (e.g., correlation or similarity) algorithm to determine whether certain objects are related. Alternatively, objects may be related by an explicit selection or designation by a user or based on a property (e.g., age, storage location, etc.) of an object. The present invention may employ some type of visual indication, such as a colored halo around thumbnails of related objects. This visual indication may be rendered continuously or, alternatively, upon an event, such as when an object thumbnail is made xe2x80x9cactivexe2x80x9d. Audio indications may also be used to indicate when a thumbnail being manipulated by a user is in close proximity to a relevant pre-existing cluster. Other audio cues may also be used to indicate other relationships between thumbnails.